ML Theory Lecture 3

– Briefly we’ll show in Theorem 1.2 (sketched last time) that the representation question is essential — sometimes, by choosing the wrong function class, you can get a bad error, even though there are simple predictors that get 0 error. – Continuous fit I: box approximations; namely, we’ll show that a variety of functions can approximate continuous functions by showing they approximate piecewise constant functions. – Continuous fit II: polynomial approximations; we’ll show refined results for other functions by showing that they in turn can be related to polynomials. – Succinct representations; recall that approximating continuous functions was “giving up”. In this subsection we’ll show that neural networks can efficiently approximate a variety of interesting functions. – GANs: lastly we’ll discuss the ability of neural networks to approximate probability distributions.